ArcnuntahMity  *  Integrity  *  Rp.llahility.. 


United  States  General  Accounting  Office  Resources,  Community,  and 

Washington,  DC  20548  Economic  Development  Division 


B-285479 


June  30,  2000 


The  Honorable  Fred  Thompson 
Chairman 

The  Honorable  Joseph  I.  Lieberman 
Ranking  Minority  Member 
Committee  on  Governmental  Affairs 
United  States  Senate 

Subject:  Observations  on  the  National  Science  Foundation’s  Fiscal  Year  1999 
Performance  Report  and  Fiscal  Year  2001  Performance  Plan 

As  you  requested,  we  have  reviewed  the  24  Chief  Financial  Officers  (CFO)  Act 
agencies'  fiscal  year  1999  performance  reports  and  fiscal  year  2001  performance 
plans  required  by  the  Government  Performance  and  Results  Act  (GPRA)  of  1993.  In 
essence,  under  GPRA  annual  performance  plans  are  to  establish  performance  goals 
and  measures  covering  a  given  fiscal  year  and  provide  the  direct  linkage  between  an 
agency’s  longer-term  goals  and  day-to-day  activities.  Annual  performance  reports  are 
to  subsequently  report  on  the  degree  to  which  those  performance  goals  were  met. 
This  report  contains  two  enclosures  responding  to  your  request  concerning  key 
program  outcomes  and  major  management  challenges  at  the  National  Science 
Foundation  (NSF).  Enclosure  I  provides  our  observations  on  NSF's  fiscal  year  1999 
actual  and  fiscal  year  2001  planned  performance  for  the  key  outcomes  that  you 
identified  as  important  mission  areas  for  the  agency.  These  key  outcomes  are  (1) 
discoveries  at  and  across  the  frontier  of  science  and  engineering  and  (2)  efficient  and 
effective  administration  of  research  grants.  Enclosure  II  lists  the  major  management 
challenges  facing  the  agency  that  NSF’s  Inspector  General  (IG)  identified,  the 
progress  the  agency  has  made  in  addressing  these  challenges  as  discussed  in  its  fiscal 
year  1999  performance  report,  and  the  applicable  goals  and  measures  in  the  fiscal 
year  2001  performance  plan. 


Results  in  Brief 

NSF’s  fiscal  year  1999  performance  report  demonstrates  that  the  agency  was 
generally  successful  in  achieving  the  outcome  involving  “discoveries  at  and  across 
the  frontier  of  science  and  engineering”  in  research  areas  such  as  biology,  Arctic  and 
Antarctic  research,  and  discoveries  in  how  the  young  learn.  NSF  judged  its 
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performance  as  successful  on  the  basis  of  (1)  program  reviews  by  independent 
committees  and  (2)  examples  of  outstanding  science  chosen  to  show  NSF’s 
achievements.  The  performance  report,  however,  sheds  little  light  on  the  quality  of 
the  committees'  reports,  and  NSF  officials  acknowledged  that  the  quality  of  these 
reports  varied.  As  a  result,  in  its  fiscal  year  2001  performance  plan,  NSF  cites 
numerous  shortcomings  in  the  committees’  program  evaluations,  such  as  “ratings” 
without  complete  information  or  examples  and  not  including  information  on  NSF’s 
funding  level  needed  to  judge  the  extent  of  NSF’s  role  in  achieving  these  results. 

Like  the  committees’  reports,  the  fiscal  year  1999  performance  report  also  fails  to 
describe  NSF’s  financial  role  in  the  examples  of  scientific  successes  presented. 
Despite  these  limitations,  the  performance  report  adequately  condenses  the 
committees’  numerous  program  reports,  which  in  turn  reflect  NSF’s 
accomplishments  with  regard  to  this  outcome. 

NSF  revised  its  fiscal  year  2001  performance  plan  by  combining  its  first  outcome  for 
fiscal  years  1999  and  2000  (discoveries  at  and  across  the  frontier  of  science)  and  its 
second  outcome  for  the  same  period  (connections  between  discoveries  and  their  use 
in  service  to  society)  into  a  single  outcome  (“Ideas-Discovery  at  and  across  the 
frontier  of  science  and  engineering,  and  connections  to  its  use  in  the  service  of 
society”).  NSF,  however,  fails  to  provide  a  clear  statement  of  expected  performance 
for  subsequent  comparison.  In  the  plan,  the  criteria  for  success  in  achieving  the 
revised  outcome  continue  to  be  too  general,  the  strategies  and  the  resources  for 
achieving  the  goals  are  not  clearly  discussed,  and  no  specific  links  between  the 
resources  and  the  areas  of  emphasis  are  included.  In  addition,  the  performance  plan 
does  not  provide  confidence  that  performance  information  will  be  credible  because, 
in  particular,  it  offers  little  assurance  that  the  inconsistent  quality  of  committee 
reports  will  be  effectively  remedied. 

NSF’s  fiscal  year  1999  performance  report  indicates  clear  progress  toward  achieving 
the  goals  embodied  in  the  second  key  outcome,  “administer  research  grants 
efficiently  and  effectively.”  The  outcome,  while  not  explicitly  included  among  NSF’s 
outcomes  in  its  performance  plans  for  fiscal  years  1999  and  2000,  may  be  inferred 
from  several  performance  goals  (such  as  using  merit  review  in  making  awards  and 
processing  proposals  in  a  timely  manner)  that  are  included  in  its  “investment  and 
management  process”  goais.  The  included  goals  and  measures  are  generally 
objective  and  quantifiable,  and  NSF  sets  high  standards  and  percentages  for 
achieving  most  goals  while  appropriately  setting  lower  percentages  for  others.  The 
performance  report  adequately  reflects  that  NSF  achieved  some,  but  not  all,  of  these 
goals.  For  example,  NSF  exceeded  its  goal  of  allocating  at  least  90  percent  of  its 
funds  to  merit-reviewed  projects,  but  it  failed  to  meet  its  goal  of  processing  70 
percent  of  the  proposals  within  6  months  of  receiving  them.  For  the  unmet  goals, 
however,  the  performance  report  does  not  provide  both  a  statement  of  the  reasons 
for  falling  short  of  the  performance  measures  and  a  strategy  for  attaining  them  in  the 
future.  In  general,  the  performance  report  provides  virtually  no  assurance  that  the 
information  is  credible  with  respect  to  these  goals. 

Among  the  key  changes  in  the  fiscal  year  2001  performance  plan,  NSF  expands  the 
definition  of  merit  review  by  including  the  integration  of  research  and  education  as  a 
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factor  to  be  considered  in  making  awards.  The  plan  also  sets  higher  standards  for 
certain  aspects  of  the  awards  process  and  for  increasing  reliance  on  electronic 
procedures.  In  this  regard,  the  plan  adds  a  goal  to  conduct  10  pilot  paperless 
projects  that  manage  the  competitive  review  process  in  a  totally  electronic 
environment.  For  the  new  elements  in  merit  review,  NSF  provides  a  clear  strategy  to 
achieve  its  goal.  However,  in  other  instances,  such  as  electronic  Filings,  NSF 
provides  little  useful  information  about  its  intended  strategy.  Additional  discussion 
of  the  problems  and  the  strategy  for  achieving  this  goal  would  have  been  helpful 
because  NSF’s  performance  actually  slipped  from  59  percent  in  fiscal  year  1998  to  58 
percent  in  fiscal  year  1999.  NSF’s  fiscal  year  2001  performance  plan  includes  only  a 
few  clear  statements  about  the  steps  to  be  taken  to  verify  and  validate  the  data.  In 
general,  as  with  the  performance  report,  the  fiscal  year  2001  plan  provides  virtually 
no  assurance  that  the  information  about  achieving  the  goals  in  this  plan  will  be 
credible. 

NSF’s  fiscal  year  1999  performance  report  and  fiscal  year  2001  performance  plan 
discussed  2  of  the  10  major  management  challenges  identified  by  NSF’s  IG  but  did  not 
address  directly  or  indirectly  the  other  8  challenges.  The  report  and  plan  addressed 
the  agency’s  progress  in  managing  an  effective  merit  review  system  and  implementing 
a  new  electronic  proposal  and  award  information  system.  The  IG  no  longer  considers 
these  issues  as  major  challenges  but  stressed  the  need  for  NSF  to  be  alert  to  emerging 
situations  that  could  result  in  their  becoming  a  problem.  The  IG  expressed  the  same 
view  regarding  four  of  the  remaining  eight  challenges.  The  IG  said  that  NSF  was 
taking  effective  steps  to  respond  to  these  challenges  and  did  not  need  to  include  them 
in  the  performance  report  or  future  performance  plans.  However,  the  IG  continues  to 
be  concerned  about  the  four  remaining  challenges  that  the  performance  report  did 
not  discuss.  These  challenges  are  all  part  of  a  single  larger  issue  about  the  adequacy 
of  NSF’s  strategy  and  resources  for  effective  oversight.  The  IG  focused  on  such 
problems  as  NSF’s  lack  of  a  system  to  assess  the  relative  vulnerabilities  of  its 
programs  and  the  commitment  of  NSF’s  funds  to  larger,  multidisciplinary  projects 
that  make  oversight  more  difficult.  As  such,  the  overall  issue  involving  oversight 
activities  might  be  suitable  for  inclusion  in  future  performance  plans.  NSF  may  want 
to  consider  whether  to  (1)  develop  a  clear  statement  of  the  IG’s  concerns  about 
oversight  activities,  (2)  assess  its  programs  for  their  degree  of  risk,  (3)  set  measurable 
goals  with  the  level  of  monitoring  corresponding  to  the  level  of  risk,  (4)  identify  the 
level  of  resources  needed  to  achieve  these  goals,  and  (5)  make  these  goals  and 
measures  a  clear  part  of  its  GPRA  performance  plans  in  the  future. 

Objectives,  Scope,  and  Methodology 

Our  objectives  concerning  selected  key  agency  outcomes  were  to  (1)  identify  and 
assess  the  quality  of  the  performance  goals  and  measures  directly  related  to  a  key 
outcome,  (2)  assess  the  agency’s  actual  performance  in  fiscal  year  1999  for  each 
outcome,  and  (3)  assess  the  agency’s  planned  performance  for  fiscal  year  2001  for 
each  outcome.  Our  objectives  concerning  major  management  challenges  were  to  (1) 
assess  how  well  the  agency's  fiscal  year  1999  performance  report  discussed  the 
progress  it  had  made  in  addressing  the  major  management  challenges  that  the 
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agency’s  IG  had  previously  identified  and  (2)  identify  whether  the  agency’s  fiscal  year 
2001  performance  plan  had  goals  and  measures  applicable  to  the  major  management 
challenges.  As  agreed,  in  order  to  meet  the  committee’s  tight  reporting  time  frames, 
our  observations  were  generally  based  on  the  requirements  of  GPRA,  guidance  to 
agencies  from  the  Office  of  Management  and  Budget  (OMB)  for  developing 
performance  plans  and  reports  (OMB  Circular  A-l  1,  Part  2),  previous  reports  and 
evaluations  by  us  and  others,  our  knowledge  of  NSF’s  operations  and  programs,  and 
our  observations  on  NSF’s  other  GPRA-related  efforts.  We  did  not  independently 
verify  the  information  contained  in  the  performance  report  or  plan.  We  conducted 
our  review  from  April  through  May  2000  in  accordance  with  generally  accepted 
government  auditing  standards. 


Agency  Comments  and  Our  Evaluation 

We  provided  a  draft  of  this  report  to  NSF  and  the  Office  of  the  Inspector  General  for 
their  review  and  comment.  We  met  with  NSF  officials,  including  the  Director,  Office 
of  Integrative  Activities;  the  Acting  Chief  Financial  Officer;  the  Director,  Information 
and  Resource  Management;  and  the  Inspector  General,  who  generally  agreed  with  the 
accuracy  of  the  facts  presented  in  the  draft  report.  NSF  officials,  however,  provided 
comments  relating  to  three  areas  of  concern  about  the  GPRA  process. 

•  First,  NSF  officials  said  that  the  agency’s  approach  in  this  first  year  of  reporting 
had  been  to  keep  the  report  brief,  succinct,  low  in  cost  to  taxpayers,  and  on 
schedule,  which  may  have  resulted  in  a  lack  of  detailed,  graphic,  and  descriptive 
information  needed  by  some  readers.  We  believe  that  NSF  would  not  have  to 
greatly  expand  the  report  or  its  plans  to  cover  the  key  information  that  was 
missing.  Moreover,  NSF  officials  also  indicated  that  they  were  concerned  with 
the  sequencing  of  products  because  the  information  needed  for  the  fiscal  year 
1999  GPRA  reports  is  also  essential  for  the  fiscal  year  1999  accountability  report 
and  the  performance  plans  for  fiscal  years  2000  and  2001.  Therefore,  according  to 
NSF  officials,  the  current  timing  of  GPRA  reports  and  plans  represents  a 
challenge.  We  believe  that  when  timing  is  an  issue,  NSF  should  state  clearly  what 
information  is  not  available  for  it  to  develop  its  reports  and  plans. 

•  Second,  NSF  officials  said  that  in  using  the  alternative  format  to  report  outcomes, 
it  used  examples  chosen  to  demonstrate  the  importance  of  NSF  funding  in 
achieving  those  results.  The  officials  believed  that  most  examples  reflect 
outcomes  that  are  achieved  over  the  long  term  but  are  not  linked  to  specific  dollar 
amounts.  We  believe  that  NSF’s  fiscal  year  1999  performance  report  is  still  unclear 
on  the  criteria  and  method  used  in  determining  that  its  funding  played  a  major 
role  in  achieving  these  outcomes.  We  believe  that  in  future  reports,  NSF  should 
demonstrate  its  role  in  achieving  these  outcomes  and  address  this  point  more 
clearly  in  selecting  and  using  examples. 

•  Third,  NSF  officials  said  that  the  agency  did  not  consider  including  a  discussion  of 
the  management  challenges  in  the  performance  report  appropriate  because  GPRA 
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does  not  require  it  and  because  NSF  addresses  the  management  challenges 
through  the  Federal  Manager’s  Financial  Integrity  Act  activities  and  reports  the 
results  in  its  annual  accountability  report.  In  addition,  NSF  officials  said  that  the 
internal  management  controls  committee  found  no  items  reportable  as  a  material 
weakness.  While  including  this  information  in  the  accountability  report  is 
important,  OMB  Circular  No.  A-ll  (1999)  indicates  that  “performance  goals  for 
management  problems  should  be  included  in  the  annual  plan.  . . .”  We  remain 
concerned  about  the  oversight  issues  raised  by  the  Inspector  General  and  would 
like  to  see  these  issues  addressed  in  NSF’s  performance  report  and  plans. 

The  IG  also  provided  some  technical  clarifications  that  we  incorporated  as 

appropriate. 


As  arranged  with  your  offices,  unless  you  publicly  announce  its  contents  earlier,  we 
plan  no  further  distribution  of  this  report  until  30  days  after  the  date  of  this  letter.  At 
that  time,  we  will  send  copies  to  Dr.  Rita  R.  Colwell,  Director,  National  Science 
Foundation,  and  make  copies  available  to  others  on  request. 


Please  call  me  on  (202)  512-3841  if  you  or  your  staff  have  any  questions.  Key 
contributors  to  this  report  were  Linda  L.  Harmon  and  Dennis  S.  Carroll. 


Jim  Wells 

Director,  Energy,  Resources, 
and  Science  Issues 


Enclosures  -  2 
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Observations  on  the  National  Science  Foundation’s  Fiscal  Year  1999  Actual 
Performance  and  Fiscal  Year  2001  Planned  Performance  Related  to  Key 

Outcomes 


This  enclosure  contains  our  observations  on  the  National  Science  Foundation’s  (NSF) 
fiscal  year  1999  actual  performance  and  fiscal  year  2001  planned  performance  for  key 
outcomes  identified  by  the  Senate  Governmental  Affairs  Committee  as  important 
mission  areas  for  NSF.  The  key  outcomes  for  NSF  are  (1)  “Discoveries  at  and  across  the 
frontier  of  science  and  engineering”  and  (2)  “Administer  research  grants  efficiently  and 
effectively.”  As  requested,  we  have  identified  the  goals  and  measures  directly  related  to 
a  selected  key  outcome.  Our  observations  are  organized  according  to  each  selected  key 
outcome  and  follow  the  goals  and  measures. 

Key  Agency  Outcome:  Discoveries  at  and  Across  the  Frontier  of  Science  and 
Engineering 

Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  of 
Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

Effectiveness  of  NSF-Supported  Research  Activities 

•  Fiscal  year  1999  goal:  NSF’s  awards  lead  to  important  discoveries;  new  knowledge 
and  techniques,  both  expected  and  unexpected,  within  and  across  traditional 
disciplinary  boundaries;  and  high-potential  links  across  those  boundaries. 

•  Actual:  In  fiscal  year  1999,  35  reports  by  NSF’s  Committees  of  Visitors  (COV)  and 
8  reports  by  NSF’s  Advisory  Committees  (AC)  rated  NSF  for  this  goal.  All  reports 
rated  NSF  as  successful  in  achieving  this  goal,  (goal  met) 

•  Fiscal  year  1999  goal:  An  independent  assessment  will  judge  NSF's  research 
programs  to  have  the  highest  scientific  quality  and  an  appropriate  balance  of 
projects  characterized  as  high  risk,  multidisciplinary,  or  innovative. 

•  Actual:  In  fiscal  year  1999,  all  of  the  COV  reports  indicated  that  NSF’s  programs 
have  high  scientific  quality.  Of  the  30  COV  reports  that  gave  an  opinion  on  the 
balance  of  projects  in  the  programs  under  review,  24  indicated  that  the  balance 
was  appropriate.  Six  reports  indicated  that  the  programs  under  review  did  not 
have  an  appropriate  balance,  with  a  lack  of  high-risk  projects  being  the  most 
frequent  concern.  Two  of  the  six  critical  reports  indicated  that  the  programs 
lacked  innovation.  NSF  concluded  that  the  goal  was  achieved,  (goal  met) 

GAO’s  Observations  on  NSF’s  Fiscal  Year  1999  Actual  Performance  for  the  Key  Agency 
Outcome  of  Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

Overall,  NSF  is  partially  successful  in  describing  its  achievement  of  this  outcome.  Its 
conclusions  depend  on  the  quality  of  the  COV  and  AC  reports  (which  NSF  uses  to  review 
its  programs)  and  the  examples  of  outstanding  science.  However,  the  performance 
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report  sheds  little  light  on  the  quality  of  the  COV  and  AC  reports.  NSF  developed 
guidelines  and  templates  to  promote  a  uniformly  high-quality  among  the  COV  and  AC 
reports.  Although  the  guidelines  show  considerable  effort,  NSF  officials  acknowledged 
that  the  reports  varied  in  quality,  and  NSF’s  fiscal  year  2001  plan  identifies  numerous 
shortcomings,  such  as  ratings  without  complete  information  in  the  reports. 

In  addition,  the  examples  in  the  COV  and  AC  reports  and  the  performance  report  do  not 
include  information  on  the  level  of  funding  by  NSF  and  other  sources  by  which  to  judge 
the  actual  extent  of  NSF’s  support  and  the  degree  to  which  success  can  be  attributed  to 
this  support.  For  instance,  two  of  NSF’s  key  examples  in  the  performance  report  include 
the  work  of  a  Nobel  Prize  winner  in  chemistry  and  of  a  National  Medal  of  Science 
awardee.  In  neither  case  does  the  performance  report  provide  sufficient  information  on 
funding  and  other  factors  to  judge  NSF’s  actual  role  in  these  achievements. 

Despite  these  limitations,  the  performance  report  is  partially  successful  in  justifying  its 
conclusion  about  the  achievement  of  the  performance  goals.  It  accurately  condenses  an 
extensive  series  of  reports  that  in  turn  generally  reflect  the  full  range  of  NSF’s  activities 
with  regard  to  this  outcome. 

Unmet  Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome 
of  Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

NSF  rated  itself  as  successful  in  achieving  these  two  goals  but,  in  the  case  of  the  second 
goal,  expressed  concern  about  the  appropriate  balance  of  projects  that  included  a  lack  of 
(1)  high-risk  projects  in  the  programs  under  review  and  (2)  innovation. 

GAO’s  Observations  on  NSF’s  Unmet  Fiscal  Year  1999  Performance  Goals  and  Measures 
for  the  Key  Agency  Outcome  of  Discoveries  at  and  Across  the  Frontier  of  Science  and 
Engineering 

NSF’s  performance  report  and  its  fiscal  year  2000  plan  partially  address  the  unmet  fiscal 
year  1999  performance  goals  and  measures  but  lack  a  clear  definition  of  the  problem  and 
specificity  in  the  steps  required  to  correct  it.  NSF’s  performance  report  states  that  staff 
will  be  asked  to  make  investments  that  address  this  goal  and  will  strive  to  identify  high- 
risk  projects  for  reviewers  and  evaluators  but  provides  no  additional  information  on 
these  investments  or  the  steps  to  be  taken.  The  report  states  that  NSF  is  also  looking  at 
how  to  improve  the  indicators  for  this  goal  and  how  to  provide  better  data  to  evaluators 
of  this  goal  for  fiscal  year  2000  but  offers  no  further  details. 

NSF’s  fiscal  year  2000  performance  plan  identifies  the  concern  about  appropriate 
balance  as  an  area  of  emphasis.  In  response  to  this  concern,  NSF  will  ask  all  directorate 
advisory  committees  to  examine  the  directorates’  entire  fiscal  year  2000  portfolio  of 
research  project  support  activities  to  identify  activities  they  would  characterize  as  high- 
risk,  multidisciplinary,  or  innovative  and  to  assess  the  overall  scientific  quality  and 
balance  with  respect  to  these  specific  characteristics. 
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NSF’s  fiscal  year  2001  performance  plan  continues  this  area  of  emphasis  and  provides  a 
careful  definition  of  high-risk  research  that  should  help  reviewers  in  identifying 
examples  of  research  in  this  category.  The  fiscal  year  2001  performance  plan  repeats  the 
very  general  statement  from  the  fiscal  year  2000  plan  about  reviewing  the  entire  research 
portfolio  but  provides  no  specific  descriptions  of  the  steps  needed  to  implement  this 
intention. 

Fiscal  Year  2000  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  of 
Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

NSF  maintained  the  same  performance  goals  and  measures  in  its  fiscal  year  2000 
performance  plan  except  that  it  dropped  the  previous  criteria  for  “minimally  effective” 
from  this  and  all  other  goals. 

GAO’s  Observations  on  NSF’s  Fiscal  Year  2000  Planned  Performance  for  the  Key  Agency 
Outcome  of  Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

NSF’s  performance  report  explained  the  elimination  of  the  criteria  for  “minimally 
effective.”  The  report  stated  that  there  was  little  to  be  gained  in  defining  minimally 
effective  and  that,  in  many  instances,  the  definition  was  confusing  to  evaluators. 
Therefore,  as  NSF  stated,  for  fiscal  year  2000,  NSF  will  define  only  the  “successful” 
standard. 

Fiscal  Year  2001  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  of 
Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

NSF  combines  its  first  and  second  outcome  goals  into  a  single  goal  for  fiscal  year  2001. 
This  goal  appears  as  follows:  Ideas  --  Discovery  at  and  across  the  frontier  of  science  and 
engineering  and  connections  to  its  use  in  the  service  of  society. 

NSF’s  performance  is  successful  when,  in  the  aggregate,  results  reported  in  the  period 
demonstrate  sufficient  progress  toward  realizing  the  following  four  performance  goals: 
(a)  a  robust  and  growing  fundamental  knowledge  base  that  enhances  progress  in  all 
science  and  engineering  areas;  (b)  discoveries  that  expand  the  frontiers  of  science, 
engineering,  and  technology;  (c)  partnerships  connecting  discovery  to  innovation, 
learning,  and  societal  advancement;  and  (d)  research  and  education  processes  that  are 
synergistically  coupled. 

GAO’s  Observations  on  NSF’s  Fiscal  Year  2001  Planned  Performance  for  the  Key  Agency 
Outcome  of  Discoveries  at  and  Across  the  Frontier  of  Science  and  Engineering 

NSF  does  not  provide  a  clear  statement  of  expected  performance  for  subsequent 
comparison  with  actual  performance.  The  criteria  for  success  in  achieving  the  revised 
outcome  resemble  the  previous  criteria  in  being  very  general  (e.g.,  “a  robust  and  growing 
fundamental  knowledge  base") .  The  plan  refers  briefly  to  the  scope  of  the  COV  in 
preparing  reports  but  gives  no  helpful  information  about  the  guidelines  or  templates  to 
be  used  by  these  committees.  In  addition,  the  plan  identifies  none  of  the  improvements 
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in  the  guidelines  that  NSF  officials  acknowledge  are  needed  to  avoid  the  problem  of 
uneven  quality  evident  in  the  first-year  performance  reports. 

The  plan  does  not  clearly  discuss  the  strategies  and  resources  for  achieving  the 
performance  goals.  It  lists  in  “bullet”  format 

•  Four  brief  points  about  resources, 

•  Four  slightly  more  detailed  areas  of  emphasis,  and 

•  Four  areas  of  multidisciplinary  research  with  two-  or  three-sentence  summaries  for 
areas  as  complex  as  global  change. 

Also,  it  provides  no  specific  links  between  the  resources  and  the  areas  of  emphasis  or 
the  multidisciplinary  research. 

The  plan  does  not  provide  confidence  that  performance  information  will  be  credible 
because,  in  particular,  it  offers  little  assurance  that  the  inconsistent  quality  of  committee 
reports  will  be  effectively  remedied.  The  plan  gives  a  clear  and  important  summary  of 
the  shortcomings  in  the  first  year  of  the  committee  reports  on  this  outcome.  These 
shortcomings  included,  among  others,  incomplete  data  making  a  performance  rating 
impossible,  ratings  without  complete  information,  the  varying  quality  of  the  examples 
selected,  and  inconsistencies  between  the  ratings  and  NSF’s  qualitative  scale.  The  key 
problem  with  NSF’s  summary  is  that  it  remains  entirely  retrospective  and  makes  no 
attempt  to  describe  what  will  be  done  to  address  these  shortcomings.  This  would  have 
been  an  appropriate  place  to  describe  any  revisions  to  the  guidelines  and  templates  and 
other  efforts  being  undertaken  to  ensure  uniformly  high  quality  in  the  second  year  of 
committee  reports. 
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Key  Agency  Outcome:  Administer  Research  Grants  Efficiently  and  Effectively 

Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  of 
Administering  Research  Grants  Efficiently  and  Effectively 

Among  NSF’s  investment  and  management  process  goals,  the  following  eight  are  directly 
related  to  this  outcome: 


Use  of  Merit  Review 

•  Fiscal  year  1999  goal:  At  least  90  percent  of  NSF’s  funds  will  be  allocated  to 
projects  reviewed  by  appropriate  peers  external  to  NSF  and  selected  through  a 
merit-based  competitive  process. 

•  Actual:  Ninety-five  percent  of  projects  allocated  funds  in  fiscal  year  1999  were 
merit  reviewed,  (goal  met) 


Implementation  of  Merit  Review  Criteria 

•  Fiscal  year  1999  goal:  NSF’s  performance  in  the  implementation  of  the  new  merit 
review  criteria  is  successful  when  reviewers  address  the  elements  of  both  generic 
review  criteria  (scientific  quality  and  impact)  appropriate  to  the  proposal  at  hand 
and  when  program  officers  take  the  information  provided  into  account  in  their 
decisions  on  awards;  minimally  effective  when  reviewers  consistently  use  only  a 
few  of  the  suggested  elements  of  the  generic  review  criteria  although  others  might 
be  applicable. 

•  Actual:  In  fiscal  year  1999,  a  total  of  44  reports  by  external  experts  rated  NSF  on 
their  use  of  the  new  merit  review  criteria.  NSF  was  rated  successful  in  achieving 
this  goal  by  36  of  those  reports,  (goal  largely  successful,  needs  improvement) 


Customer  Service-Time  to  Prepare  Proposals 

•  Fiscal  year  1999  goal:  Ninety-five  percent  of  program  announcements  and 
solicitations  will  be  available  at  least  3  months  prior  to  proposal  deadlines  or 
target  dates  improving  upon  the  fiscal  year  1998  baseline  of  66  percent. 

•  Actual:  Seventy-five  percent  of  program  announcements  and  solicitations  were 
available  at  least  3  months  prior  to  proposal  deadlines  or  target  dates,  (goal  not 
met) 
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Customer  Service-Time  to  Decision 

•  Fiscal  year  1999  goal:  Process  70  percent  of  proposals  within  6  months  of  receipt 
improving  upon  the  fiscal  year  1998  baseline  of  59  percent. 

•  Actual:  Fifty-eight  percent  of  proposals  were  processed  within  6  months  of 
receipt,  (goal  not  met) 

Electronic  Proposal  Processing 

•  Fiscal  year  1999  goal:  NSF  will  receive  and  process  at  least  25  percent  of  full 
proposal  submissions  electronically  through  FastLane,  improving  upon  the  fiscal 
year  1998  baseline  of  17  percent. 

•  Actual:  In  fiscal  year  1999,  44  percent  of  full  proposal  submissions  were  received 
through  FastLane.  (goal  met) 


Capability  in  Use  of  Information  Technology 

•  Fiscal  year  1999  goal:  By  fiscal  year  1999,  all  staff  will  receive  an  orientation  to 
FastLane,  and  at  least  95  percent  of  program  and  program  support  staff  will 
receive  practice  in  using  its  key  modules. 

•  Actual:  Eighty  percent  of  all  employees  received  an  orientation  to  FastLane  and 
40  percent  of  program  and  program  support  staff  received  practice  in  using  its  key 
modules,  (goal  not  met) 

Year  2000 

•  Fiscal  year  1999  goal:  NSF  will  complete  all  activities  needed  to  address  the  Year 
2000  problem  for  its  information  systems  according  to  plan,  on  schedule,  and 
within  budget  during  fiscal  year  1999. 

•  Actual:  NSF  said  that  it  achieved  this  goal,  (goal  met) 


Project  Reporting  System 

•  Fiscal  year  1999  goal:  During  fiscal  year  1999,  at  least  70  percent  of  all  project 
reports  will  be  submitted  through  the  electronic  Project  Reporting  System. 

•  Actual:  Fifty-nine  percent  of  all  projects  reports  were  submitted  through  the 
electronic  Project  Reporting  System,  (goal  not  met) 
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GAO’s  Observations  on  NSF’s  Fiscal  Year  1999  Actual  Performance  for  the  Key  Agency 
Outcome  of  Administering  Research  Grants  Efficiently  and  Effectively 

NSF  has  not  created  a  specific  mission  outcome  with  respect  to  administering  research 
grants  efficiently  and  effectively.  This  mission  outcome  may  be  inferred  from  eight  of 
the  performance  goals  included  in  NSF’s  investment  and  management  process  goals. 
These  eight  goals  contribute  to  measuring  such  an  outcome  but  may  not  be  the  only 
issues  needed  to  be  considered  in  a  comprehensive  assessment  of  grant  administration. 

NSF  discusses  clear  progress  toward  achieving  the  performance  goals.  The  measures  set 
high  standards  and  percentages  for  achieving  most  goals  (such  as  the  use  of  merit 
review)  and  appropriately  lower  percentages  for  others  (such  as  the  use  of  FastLane  [a 
new  electronic  proposal  and  award  information  system]  in  the  goal  for  Electronic 
Proposal  Processing) .  The  goals  and  their  measures  are  objective  and  quantifiable  with 
the  exception  of  the  goal  for  Implementation  of  Merit  Review  Criteria,  which  used  the 
OMB-approved  alternative  format. 

The  performance  report  clearly  articulates  the  degree  to  which  NSF  achieved  its  goals, 
with  precise  percentages  showing  how  far  NSF  exceeded  or  fell  short  of  its  targets.  For 
example,  NSF  exceeded  the  goal  for  Use  of  Merit  Review  by  5  percent  and  fell  short  of 
its  goal  for  Customer  Service-Time  to  Prepare  Proposals  by  20  percent.  In  the  case  of 
the  goal  for  Implementation  of  Merit  Review  Criteria,  the  one  nonquantitative  goal 
among  these  eight  goals,  the  report  clearly  discusses  the  COV  and  AC  reports  from  which 
NSF  drew  its  conclusion  that  the  agency  was  largely  successful. 

Despite  this  apparent  precision,  NSF  provides  virtually  no  assurance  that  the 
performance  information  is  credible  with  respect  to  these  goals.  The  report  frequently 
states  the  absence  of  data-related  problems  but  does  not  describe  the  steps  taken  to 
verify  and  validate  the  data.  For  the  goals  including  Use  of  Merit  Review,  Customer 
Service-Time  to  Prepare  Proposals,  Customer  Service-Time  to  Decision,  Electronic 
Proposal  Processing,  Capability  in  Use  of  Information  Technology,  and  Year  2000,  the 
report  indicates  that  data  limitations  were  generally  nonexistent  or,  in  one  case,  early 
issues  were  corrected.  For  the  goal  Project  Reporting  System,  the  report  states  that 
paper  copies  of  reports  not  captured  electronically  are  not  counted  but  does  not  indicate 
the  extent  to  which  this  might  influence  the  data.  For  the  goal  Implementation  of  Merit 
Review  Criteria,  the  report  noted  that  adequate  data  were  not  always  available. 

The  limitations  of  the  report  in  terms  of  verification  and  validation  are  especially  evident 
in  the  presentation  of  NSF’s  Data  Quality  Project.  In  a  one-page  overview  of  data 
verification  and  validation  activities,  the  report  lists  six  objectives  associated  with  the 
Data  Quality  Project  for  the  Investment  and  Management  goals,  including  these  eight 
goals.  The  first  objective,  for  example,  is  to  evaluate  the  quality  of  the  data  in  the  central 
databases  at  NSF.  The  report  provides  no  additional  information  about  the  Data  Quality 
Project,  its  six  objectives,  or  the  status  of  its  activities. 

Unmet  Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome 
of  Administering  Research  Grants  Efficiently  and  Effectively 
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According  to  the  performance  report,  the  goals  for  Customer  Service-Time  to  Prepare 
Proposals,  Customer  Service-Time  to  Decision,  Capability  in  Use  of  Information 
Technology,  and  Project  Reporting  System  were  not  achieved,  and  the  results  for  the 
goal  Implementation  of  Merit  Review  Criteria,  while  largely  successful,  needed  some 
improvement. 

GAO’s  Observations  on  NSF’s  Unmet  Fiscal  Year  1999  Performance  Goals  and  Measures 
for  the  Key  Agency  Outcome  of  Administering  Research  Grants  Efficiently  and 
Effectively 

NSF  does  not  provide  both  a  statement  of  the  reasons  for  falling  short  of  these  goals  and 
a  strategy  to  attain  them.  In  general,  the  report  does  not  explain  why  NSF  failed  to  attain 
the  goals  but  does  provide  a  strategy  for  attaining  them  in  the  future.  For  the  goal 
Project  Reporting  System,  however,  the  report  explains  the  reasons  for  not  attaining  it 
but  provides  no  plan  for  doing  so  in  the  future. 

More  specifically,  for  the  goal  Implementation  of  Merit  Review  Criteria,  which  was 
largely  successful,  the  report  does  not  explain  the  reasons  for  shortcomings  in  some 
cases  but  notes  that  NSF  has  re-issued  guidance  to  applicants  and  reviewers,  stressing 
the  importance  of  using  both  criteria  (scientific  quality  and  impact)  in  preparing  and 
evaluating  proposals  submitted  to  NSF. 

For  the  goal  Customer  Service-Time  to  Prepare  Proposals,  NSF  plans  to  determine  why 
the  goal  was  not  met.  The  report  also  notes  that  a  Web-based  system  for  creating 
program  announcements  has  been  established  and  is  expected  to  aid  in  achieving  this 
goal. 

For  the  goal  Customer  Service-Time  to  Decision,  NSF  does  not  explain  why  the  goal  was 
not  met.  NSF  plans  to  shorten  the  award-processing  time  by  making  more  effective  use 
of  electronic  mechanisms  and  carefully  tracking  the  processing  of  all  proposals. 
Additional  discussion  of  the  problems  and  strategies  for  achieving  this  goal  would  have 
been  helpful  because  NSF’s  performance  actually  slipped  from  59  percent  in  fiscal  year 
1998  to  58  percent  in  fiscal  year  1999. 

For  the  goal  Capability  in  Use  of  Information  Technology,  NSF  does  not  explain  why  the 
goal  was  not  met  but  notes  that  staff  will  be  strongly  encouraged  to  attend  orientation 
and  training  sessions.  While  the  performance  report  makes  no  additional  observations, 
NSF’s  fiscal  year  1999  accountability  report  notes  that  the  most  common  reason  for  not 
attending  orientation  and  training  sessions  was  scheduling  conflicts.  In  fiscal  year  2000, 
the  number  of  class  offerings  will  be  increased  to  ensure  that  the  goal  is  met. 

For  the  goal  Project  Reporting  System,  NSF  provides  a  clear  explanation  of  the  reasons 
why  it  missed  the  goal,  including  transition  to  a  new  reporting  system  and  granting 
permission  to  scientists  submitting  proposals  to  use  the  old  paper  report  forms  during 
the  first  3  months  of  the  fiscal  year.  NSF  states  that  the  new  system  is  being  modified 
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and  enhanced  but  does  not  provide  details  about  these  improvements  or  any  other  steps 
to  meet  the  goal. 

Fiscal  Year  2000  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  of 
Administering  Research  Grants  Efficiently  and  Effectively 


Electronic  Proposal  Processing 

•  Fiscal  year  2000  goal:  NSF’s  plan  raises  its  performance  measure  from  25 
percent  to  60  percent  for  submitting  proposals  electronically,  (goal  revised) 


Capability  in  Use  of  Information  Technology 

•  Fiscal  year  2000  goal:  NSF’s  plan  maintains  its  performance  measure  at  100 
percent  for  orientation  to  FastLane  but  reduces  its  performance  measure  from 
95  to  80  percent  for  program  and  program  support  staff  to  receive  training. 

(goal  revised) 

Project  Reporting  System 

•  Fiscal  year  2000  goal:  This  goal  has  a  new  performance  measure  for  eligible 
project  reports  to  be  submitted  through  the  electronic  Project  Reporting 
System.  NSF  has  raised  the  goal  to  85  percent,  (goal  revised) 


GAQ’s  Observations  on  NSF’s  Fiscal  Year  2000  Planned  Performance  for  the  Key  Agency 
Outcome  of  Administering  Research  Grants  Efficiently  and  Effectively 

For  the  goal  Electronic  Proposal  Processing,  NSF’s  fiscal  year  1999  goal  was  25  percent, 
and  fiscal  year  1999  results  showed  that  44  percent  of  the  proposals  were  submitted 
electronically.  NSF  does  not  provide  a  clear  explanation  for  setting  the  increased  goal 
at  60  percent  in  fiscal  year  2000.  NSF  said  that  the  goal  will  be  met  through  increasing 
the  user  friendliness  of  the  system,  programs  requiring  submission  through  FastLane, 
stronger  efforts  to  encourage  such  submission,  enhanced  staff  training,  and  additional 
resources  requested. 

For  the  goal  Capability  in  Use  of  Information  Technology,  NSF’s  plan  does  not  explain 
the  reduction  from  95  to  80  percent  as  a  goal  for  training  staff  in  FastLane.  NSF’s 
discussion  of  this  goal  is  very  brief  and  provides  little  specific  information  on  how  it  will 
be  achieved. 

For  the  goal  Project  Reporting  System,  NSF’s  fiscal  year  1999  goal  was  70  percent,  and 
fiscal  year  1999  results  showed  that  59  percent  of  project  reports  were  submitted 
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through  the  new  Project  Reporting  System.  NSF  does  not  provide  a  clear  explanation  for 
setting  the  increased  goal  at  85  percent  in  fiscal  year  2000.  NSF  said  that  the  goal  will  be 
met  through  revisions  to  a  relevant  manual  and  issuance  of  a  notice  to  grantees  requiring 
that  all  project  reports  be  submitted  electronically  in  fiscal  year  2000. 

Fiscal  Year  2001  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  of 
Administering  Research  Grants  Efficiently  and  Effectively 


Implementation  of  Merit  Review  Criteria 

•  Fiscal  year  2001  goal:  This  goal  includes  a  new  criterion-integration  of 
research  and  education  in  reviews.  In  addition,  in  a  pilot  effort,  at  least  50 
percent  of  research  grant  proposals  will  address  this  new  element  along  with 
previous  elements,  (goal  revised) 


Customer  Service-Time  to  Decision 

•  Fiscal  year  2001  goal:  This  goal  increases  its  performance  measure  from  70 
percent  to  75  or  80  percent  (see  discussion  of  this  discrepancy)  for  processing 
of  proposals  within  6  months  of  receipt,  (goal  revised) 


Electronic  Proposal  Processing 

•  Fiscal  year  2001  goal:  This  goal  has  a  new  performance  measure,  increasing  to 
95  percent  for  submission  of  proposals  electronically,  (goal  revised) 


Year  2000 

•  Fiscal  year  2001  goal:  This  goal,  involving  Year  2000  issues,  has  been  achieved 
and  omitted  from  the  fiscal  year  2001  plan,  (goal  discontinued) 

Project  Reporting  System 

•  Fiscal  year  2001  goal:  This  goal  has  a  new  performance  measure,  increasing  to 
95  percent  for  eligible  project  reports  to  be  submitted  through  the  electronic 
Project  Reporting  System,  (goal  revised) 

In  addition  to  these  modifications  to  existing  goals,  NSF  has  added  two  new  goals  that 
relate  to  administering  research  grants  efficiently  and  effectively. 
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Electronic  Proposal  Processing 

•  Fiscal  year  2001  goal:  NSF  will  conduct  10  pilot  paperless  projects  that 
manage  the  competitive  review  process  in  a  totally  electronic  environment. 

(new  goal) 


Work  Environment 

•  Fiscal  year  2001  goal:  NSF  will  strive  to  provide  NSF  staff  with  a  physical 
environment  that  is  safe  and  well-equipped  with  current  technology  tools.  The 
remainder  of  this  goal,  as  stated  in  the  plan,  does  not  appear  directly  relevant 
to  administering  grants  efficiently  and  effectively,  (new  goal) 

GAO’s  Observations  on  NSF’s  Fiscal  Year  2001  Planned  Performance  for  the  Key  Agency 
Outcome  of  Administering  Research  Grants  Efficiently  and  Effectively 

For  the  goal  Implementation  of  Merit  Review  Criteria,  the  report  clearly  states  NSF’s 
intention  of  making  the  integration  of  research  and  education  a  key  part  of  the  proposals 
and  the  agency’s  review  of  these  proposals.  The  report  also  includes  a  clear  statement  of 
its  actions  to  achieve  this  goal,  including  modifying  its  program  announcements  and 
reissuing  guidance  to  applicants  and  reviewers  that  stresses  the  importance  of  this  new 
criterion.  NSF  will  use  the  directorate  advisory  committees  to  monitor  this  goal  and  will 
ask  them  to  describe  in  their  reports  the  samples  and  evidence  supporting  their 
conclusions  about  its  attainment.  This  is  one  of  the  few  specific  statements  in 
conjunction  with  a  goal  about  activities  to  enhance  the  credibility  of  the  information. 

For  the  goal  Customer  Service-Time  to  Decision,  NSF’s  fiscal  year  2001  plan  states  the 
performance  goal  as  75  and  80  percent  on  the  same  page.  The  correct  goal  cannot  be 
determined  from  the  context.  NSF’s  statement  of  its  strategy  repeats  the  agency’s  fiscal 
year  2000  plan.  Additional  discussion  of  the  problems  and  strategies  for  achieving  this 
goal  would  have  been  helpful  because  NSF’s  performance  actually  slipped  from  59 
percent  in  fiscal  year  1998  to  58  percent  in  fiscal  year  1999.  The  plan  states  that  no  data 
limitations  exist  in  verifying  this  information,  but  the  absence  of  a  clear  discussion  of  the 
Data  Quality  Project  diminishes  confidence  in  the  credibility  of  the  data. 

For  the  goal  Electronic  Proposal  Processing,  our  comments  on  this  goal  in  the  fiscal  year 

2000  plan  hold  true  for  the  new  goal  of  95  percent.  The  only  addition  to  the  fiscal  year 

2001  plan  is  a  reference  to  establishing  a  Helpdesk  to  assist  external  customers. 

For  the  goal  Project  Reporting  System,  NSF  clearly  states  its  intention  of  reaching  95 
percent  for  project  reports  to  be  submitted  through  the  Project  Reporting  System.  The 
report  stresses  the  importance  of  moving  quickly  to  reach  this  goal  so  the  database 
represents  the  outputs  and  outcomes  of  NSF-funded  activities.  The  report  repeats  the 
strategy  stated  in  the  fiscal  year  2000  plan  for  achieving  this  goal.  To  enhance  the 
credibility  of  the  information,  the  Division  of  Information  Systems  and  the  Budget 
Division  will  jointly  monitor  the  Project  Reporting  System  and  data  on  its  use.  This  is 


16 


GAO/RCED-00-205R  NSF’s  FY  1999  Performance  Report  and  FY  2001  Performance  Plan 


Enclosure  I 


another  helpful  statement  indicating  a  specific  effort  that  should  strengthen  the 
credibility  of  the  information. 

For  the  new  goal  Electronic  Proposal  Processing,  NSF  states  its  goal  clearly  and  adds 
that,  during  fiscal  year  2000,  NSF  was  to  address  the  barriers  to  achieving  this  goal  but 
does  not  describe  the  efforts  to  address  these  barriers.  NSF  notes  that  many  reviewers 
are  not  comfortable  with  electronic  proposals,  and  it  may  be  important  to  generate  hard 
copies  for  reviewers  who  want  them.  This  caveat  appears  to  undermine  NSF’s 
determination  to  achieve  the  goal. 

For  the  new  goal  Work  Environment,  NSF’s  new  goal  appears  to  address  the  technology 
involved  in  FastLane  under  the  previous  goal  for  Capability  in  Use  of  Information 
Technology.  However,  NSF  makes  no  specific  reference  to  FastLane  in  its  fiscal  year 
2001  plan,  although  training  in  this  technology  was  one  of  the  goals  of  NSF’s  fiscal  year 
2000  plan.  The  new  goal,  as  stated,  is  very  general  and  contains  no  performance 
measures.  In  addition,  the  plan  contains  no  reference  to  any  progress  toward  achieving 
the  FastLane  goal  of  training  80  percent  of  staff  to  use  this  technology. 
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